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Implementation and Early Outcomes of 
the Comprehensive School Reform Demonstration Program 



Executive Summary 



Created in 1998 under Public Law 105-78, and authorized under the No Child Left 
Behind (NCLB) Act of 2001, the Comprehensive School Reform (CSR) program provides 
financial assistance to help schools develop and implement systematic approaches to schoolwide 
improvement that are grounded in scientifically based research and effective practices. The goal 
of the program is to enable all children to meet challenging state academic content and 
achievement standards. The annual grants of at least $50,000 per school support the initial 
implementation costs of adopting a research based refonn strategy, over a three-year period. 
Since the program’s inception in 1998, federal appropriations totaling nearly $1.4 billion have 
supported grants to over 5,000 recipients. 

The federal CSR program builds on the research on effective schools and expands the 
concept of the Title I “schoolwide program,” first introduced in the 1988 Hawkins-Stafford 
amendments to the Elementary and Secondary Education Act (ESEA). Before 1988, federal 
funding for low-income and low-performing schools was to provide targeted services to students 
on an individualized basis. Schoolwide programs allow high-poverty schools to use federal 
resources in a comprehensive, integrated way to reform the entire school to meet the educational 
needs of all students in the school. The CSR program, targeted to schools serving the same high- 
poverty student populations, provides additional resources to help schools implement a cohesive 
reform plan. 

Under the provisions of the federal CSR program, schools may implement any reform 
strategy that is based on rigorous research, within a plan that addresses 1 1 components of 
comprehensive school reform detailed in the No Child Left Behind Act (Exhibit E-l). While the 
original legislation included nine components of reform, the 200 1 legislation added the 
importance of support for school staff and stressed the need for scientifically based research to 
improve academic outcomes of students in participating schools; NCLB also removed the term 
“demonstration” from the program’s name. Throughout this report, descriptions of the 
evaluation findings, which relate to the program as created in 1998, will refer to the “CSRD” 
program, while descriptions of the current program authorized in the 2001 legislation will refer 
to the “CSR” program. 

National Evaluation of the Federal CSRD Program 

The 1998 legislation also mandated national evaluation activities, requiring the 
Department of Education “to assess results achieved by the implementation of comprehensive 
school reform in Title I schools.” The Department released its first report in 2000, describing 
early implementation of the CSRD program. This report provides updated implementation data 
and analyzes preliminary data on achievement outcomes for CSRD schools. 
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Exhibit E-l 

Eleven Components of Comprehensive School Reform 
Described in the No Child Left Behind Act 



■ Proven methods and strategies for student learning, teaching, and school management that are based on 
scientifically based research and effective practices and that have been replicated successfully in schools with 
diverse characteristics. 

■ Comprehensive design for effective school functioning, integrating instruction, assessment, classroom 
management, and professional development and aligning these functions into a schoolwide reform plan 
designed to enable all students to meet challenging state content and performance standards and address needs 
identified through a school needs assessment. 

■ Professional development. High-quality and continuous teacher and staff professional development and 
training. 

■ Measurable goals for student performance and benchmarks for meeting those goals. 

■ Support from staff. Support from school faculty, administrators, and staff. 

■ Support for staff. Support for school faculty, administrators, and staff. (Added in 2001) 

■ Parent and community involvement. Meaningful involvement of parents and the local community in 
planning and implementing school improvement activities. 

■ External assistance. High-quality external support and assistance from a comprehensive school reform entity 
(which may be a university) with experience in schoolwide reform and improvement. 

■ Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

■ Coordination of resources. Identification of how other available resources (federal, state, local, or private) 
will help the school coordinate services to support and sustain the school reform. 

■ Scientifically based research. Scientifically based research to significantly improve the academic achievement 
of students participating in such programs as compared with students in schools who have not participated in 
such programs. This requirement may also be met by strong evidence that such programs will significantly 
improve the academic achievement of participating children. (Added in 2001) 

Source: Elementary and Secondary Education Act as amended by the No Child Left Behind Act, Title I, Part F, 

Section 1606. 



Guided by the components included in the authorizing legislation, the National 
Evaluation of the CSRD Program addresses the following research questions: 

• Are CSRD funds well targeted to schools with the greatest need? 

• Have schools implemented the nine components of CSRD? 

• Has student achievement improved in funded schools? 
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Data to address these three research questions were obtained from four sources: the 
National Longitudinal Survey of Schools (NLSS), the National School-Level State Assessment 
Score Database, the Field-Focused Study of the CSRD Program, and the Southwest Educational 
Development Laboratory (SEDL) database of CSRD grantee information. Data presented in this 
report are for various years between 1998-99 and 2001-02. 



Key Findings 

CSRD funds are well targeted. 

CSRD funds are more likely to be received by schools with a high concentration of 
students who are low income, minority, and limited English proficient (Exhibit E-2). Nearly half 
(46 percent) of CSRD schools were in the highest poverty category (defined as having more than 
75 percent of their students eligible for free or reduced-price lunches), compared with only 17 
percent of all schools in the United States. 

CSRD schools are also more likely to be iden tified as in need of improvemen t under 
Title I (37 percent) compared with Title I schools (17 percent) or all schools (9 percent). CSRD 
grants are concentrated in urban settings and elementary schools. 



Exhibit E-2 

Demographic Characteristics of CSRD Schools, 1999-2000 





CSRD 


All 




Schools 


Schools 


Percent of students eligible for free or reduced-price lunches 






75-100% 


46%* 


17% 


50-74.9% 


31%* 


19% 


35-49.9% 


9%* 


16% 


0-34.9% 


14% * 


47% 


Percent of schools identified as in need of improvement 


37%* 


9% 


under Title I 






Percent of minority students 






75-100% 


46%* 


16% 


50-74.9% 


1 8%* 


12% 


25-49.9% 


14% 


16% 


0-24.9% 


22% * 


55% 


Percent of students with limited English proficiency (LEP) 






25-100% 


15%* 


7% 


10-24.9% 


14%* 


8% 


0. 1-9.9% 


27%* 


39% 


0% 


44% 


46% 



Exhibit reads: The highest-poverty group of schools, where 75 percent or more of 
the students are eligible for free or reduced-price lunches, accounted for 46 percent of 
CSRD schools and 17 percent of all schools. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 
level. 



Source: NLSS Principal Survey, 1999-2000, and Schools and Staffing Survey, 1999-2000. 




CSRD schools are more likely to adopt external reform methods, and staff in CSRD schools 
showed greater support for the school’s chosen reform method. In a number of other 
areas, CSRD schools and non-CSRD schools did not differ significantly, although this was 
often because non-CSRD schools as well as CSRD schools were highly likely to report 
practices associated with comprehensive school reform. 

In 2000-01, every CSRD school reported having adopted an externally developed refonn 
method, compared with 71 percent of non-CSRD schoolwide schools (Exhibit E-3). CSRD 
schools showed greater support from school staff for the school’s reform method. In particular, 
teachers were more likely to report that the school’s reform method had improved teaching, 
professional growth, and students’ engagement in learning “to a great extent.” 

Teachers in CSRD and non-CSRD schoolwides were about equally likely to report 
receiving professional development (four-fifths of teachers in both groups), but teachers in 
CSRD schools reported fewer total hours of professional development than teachers in non- 
CSRD schoolwide schools. Findings varied by school level; elementary school teachers in 
CSRD schools were more likely to obtain professional development in reading and math than 
their counterparts in non-CSRD schools, but no significant difference was found for secondary 
school teachers. Principals in CSRD schools were more likely to report that professional 
development activities in their school were influenced by school plans, student assessment data, 
and state or local content standards. 

On most other measures — comprehensive planning, measurable goals, parent and 
community involvement, and evaluation — the NLSS surveys did not find significant differences 
between CSRD and non-CSRD schoolwide programs. Nearly all schools — both CSRD and non- 
CSRD schoolwides — reported having a written comprehensive or strategic plan, quantifiable 
annual goals for student achievement, and a student assessment component to their primary 
reform method. Teacher reports on parent involvement were similar in both groups of schools. 
No significant difference was found between CSRD and non-CSRD schools on combining 
federal funds with other funding sources, although CSRD principals were less likely to report 
uncertainty about what is allowed in this area. 

Case studies in 18 sites indicate that implementation of the nine CSRD components was 
mixed. 



While half of the 18 CSRD schools were fully implementing their chosen method, 
implementation was uneven or faltering in one-third of the schools, and at three of the schools, 
implementation was minimal or had stalled completely. In addition, fidelity to the original 
method, as judged by the model developer, was considered “high” in less than half of the 
schools. 

Nearly all of the case study schools were engaged in moderate to intensive professional 
development, and most of this professional development was provided through an external 
technical support team. The professional development was largely focused on implementation of 
the model program, not necessarily comprehensive refonn more broadly. 
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Exhibit E-3 

Implementation of Comprehensive School Reform 
in CSRD and Non-CSRD Schools with Schoolwide Programs 





CSRD 


Non-CSRD 




Schools 


Schools 


Innovative strategies and methods 

Adoption of externally developed reform method 


100%* 


71% 


Percent of teachers who participated in 
professional development in the past year 

Professional development in their content area 


82% 


79% 


Average number of PD hours in content areas 


28 hours* 


34 hours 


Important influences on professional development, as reported by principals 

School plans 87%* 


72% 


Student assessment data 


78%* 


65% 


Implementation of content standards 


74%* 


62% 


Comprehensive plan for school reform 

Overall written annual or strategic plan in SY 2000-01 


99% 


99% 


Measurable goals and benchmarks 

Quantifiable annual goals for student achievement 


96% 


89% 


Adopted reform methods with student assessments, goals, 
and benchmarks 


90% 


93% 


Support from school staff 

Teachers report supporting reform method “to a great extent” 
Teachers report that reform method improved teaching, professional 
growth, and students engagement in learning “to a great extent” ** 


61% 

40% - 50%* 


55% 

33% -41% 


Parent involvement 

Teachers report using school-parent compacts 
Teachers discussed how teachers and parents could help 
the student achieve these goals 


64% 

98% 


64% 

99% 


Evaluation 

Primary reform method included a student assessment component 


95% 


98% 


Coordination of resources 

Combined federal funds with other funding sources 


52% 


44% 


“to a great extent” 

Principals uncertain about what is allowed 


31%* 


41% 



Exhibit reads: All CSRD schoolwide schools reported adopting an externally developed reform method, 
compared with 71 percent of non-CSRD schoolwides. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 

** This item combines responses from three questions: 1) impact on teaching, 2) professional growth, and 
3) students’ engagement. The range of percentages reflects the answers across all three questions. 

Source: NLSS Teacher and Principal Surveys, 1998-99, 1999-2000, and 2000-01. 



IX 










Less than half of the case study schools had in place reforms that were judged to be 
comprehensive. Some of the schools had comprehensive plans, but staff were unaware of or 
were detached from any comprehensive vision associated with these plans; in other schools, 
there was neither documentation nor staff understanding that demonstrated that the school had a 
comprehensive plan for improving the school. Most model programs themselves are by design 
not comprehensive because they are limited to certain grades or certain academic subjects. 

All but one of the schools had measurable goals and benchmarks for measuring progress 
in student achievement. Schools typically did not measure their progress in implementing the 
nine CSRD components. 

High turnover among staff and students at some schools resulted in a transient level of 
staff and parent support for the reform effort. Few schools had defined strategies to produce 
broad and sustained parental and community involvement. 

Most of the case study schools were coordinating funds to support the reform effort, 
although some schools were coordinating only funds from federal sources because they did not 
have the access to local funds needed to redirect them. Nearly all of the schools were unclear 
about the amount or source of new funds that might be used to implement the research-based 
method after the CSRD award has ended. 

Early evidence suggests that some CSRD schools made gains on state assessments, but no 
relationship was found between CSRD funding and improved student achievement. 
However, these findings should be viewed with caution because the achievement data 
available for this report covered a very short period of implementation. 

Although this report includes an analysis of student achievement data , findings regarding 
student outcomes are preliminary due to the limited amount of data available for this report. 
Trends in state assessment results were examined over a three-year period, from 1998-99 through 
2000-01, in 38 states that had this data available; however, for about two-thirds of these states, 
only two years of data were available. (Data showing whether absolute gains occurred were 
available in only 28 states, while data showing relative gains were available for 38 states.) 

Not only is this period of time very short for making reasonable judgments about 
achievement effects, the amount of time that schools had to implement their CSRD grants may 
have been even shorter due to delays in some states awarding CSRD grants. For these reasons, 
the analyses presented here have limited value forjudging the effects of the CSRD program on 
student achievement. 
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Overall, CSRD schools made gains on state assessments in reading and mathematics in 
about one-quarter of the states. Among elementary schools, CSRD schools made gains in 
reading in 10 out of 28 states; they made gains in mathematics in 9 out of 27 states (Exhibit E-4). 
Achievement gains were less common at the middle school and high school levels. 

Exhibit E-4 

Summary of State-Level Changes in CSRD Schools 
From Baseline to One to Two Years after Receipt of CSRD Funds 





Number 
of States 


Positive 


No Change* 


Negative 


Elementary Reading 


28 


10 


14 


4 


Elementary Math 


27 


9 


18 


0 


Middle School Reading 


23 


4 


18 


1 


Middle School Math 


23 


4 


18 


1 


High School Reading 


14 


1 


11 


2 


High School Math 


14 


3 


11 


0 


Total 


129 


31 


90 


8 


% of Cases 




24% 


70% 


6% 



* Includes states where change was not statistically significant. Statistical significance was 
evaluated at the .05 level of confidence using a two-sample t-test. 



Exhibit reads: State reading assessments for elementary schools showed positive growth for 
CSRD schools in 10 out of 28 states; negative changes occurred in 4 states. 



However, states with significant improvement in student achievement for CSRD schools 
also had significant growth in achievement for non-CSRD schools (Exhibit E-5). Looking at 
change in the within-state rankings of CSRD schools compared with all other schools in the 
state, more states show positive average gains than declines, particularly at the elementary level 
(Exhibit E-6). These gains are slightly more prevalent in states that have assessment trend data 
available for a longer period of time (three years instead of the usual two years), raising hope for 
further analyses when additional years of data are available. However, nearly all of the gains 
(and losses) were not statistically significant. 

This analysis of student outcomes should be considered preliminary because the 
time frame covered here is too short to expect large measurable effects of the CSRD 
program. Most schools had been receiving CSRD funding for less than one or two years, and 
the state assessment data that was available provided only one or two years of achievement 
change data. Additional time would allow for deeper implementation of reforms. Additional 
time would also provide increased data points, allowing for a more robust analysis. A later 
report on student achievement in the first cohort of CSRD grantees would offer a better 
understanding of the progress of schools participating in this program. 
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Exhibit E-5 

States with Significant Growth in Achievement for CSRD Elementary Schools 







Reading 


Mathematics 


CSRD 

Schools 


Non-CSRD 

Schools 


CSRD 

Schools 


Non-CSRD 

Schools 


States with One-Year Change Data (1998-99 to 1999-2000) 


Colorado 




+ 


+ 


NS 


NS 


Minnesota 




+ 


+ 


+ 


+ 


South Carolina 


+ 


+ 


+ 


+ 



States with Two-Year Change Data (1998-99 to 2000-01) 



California 


+ 


+ 


+ 


+ 


Florida 


+ 


+ 


+ 


+ 


Kentucky 


+ 


+ 


+ 


+ 


Massachusetts 


+ 


+ 


NS 


NS 


New York 


+ 


+ 


+ 


+ 


Texas 


+ 


+ 


+ 


+ 


Virginia 


NS 


NS 


+ 


+ 


Washington 


+ 


+ 


+ 


+ 



* Statistical significance was evaluated at the .05 level of confidence using a two-sample t-test. 
“NS” indicates that the change in achievement was not statistically significant. 



Exhibit reads: In Colorado, CSRD schools made significant gains on state reading assessments 
from 1998-99 to 1999-2000, but non-CSRD schools also had significant achievement growth. 



Exhibit E-6 

Summary of Change in Within-State Rankings for CSRD Schools 





All States 


States with Two Years of Change Data 


Number 
of States 


Positive 


No 

Change 


Negative 


Number 
of States 


Positive 


No 

Change 


Negative 


Elementary Reading 


38 


21 


1 


16 


9 


6 


0 


3 


Elementary Math 


37 


22 


2 


13 


9 


6 


1 


2 


Middle School Reading 


29 


15 


0 


14 


8 


3 


0 


5 


Middle School Math 


29 


15 


0 


14 


8 


4 


0 


4 


High School Reading 


15 


7 


0 


8 


6 


3 


0 


3 


High School Math 


15 


9 


1 


5 


6 


4 


2 


1 


Total 


163 


89 


4 


70 


46 


26 


2 


18 


% of Cases 




55% 




43% 




57% 




39% 



Exhibit reads: Across all states, subjects, and grade levels, CSRD schools made gains relative to non-CSRD 
schools in 55 percent of the states, on average, and relative declines in 43 percent of the states; however, 
very few of these changes were statistically significant. 
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Section I: The Federal Role in Comprehensive School Reform 



Highlights 

Public Law 105-78. The Comprehensive School Reform Demonstration (CSRD) program was initially 
created in 1998 under Public Law 105-78. It was authorized as the Comprehensive School Reform 
(CSR) program in the Elementary and Secondary Education Act as amended by the No Child Left Behind 
Act of 2001. 

Varied federal role. The U.S. Department of Education supports comprehensive school reform through 
a state formula grant program, technical assistance, and research and development efforts to support the 
development of research-based reform methods. 

Funding. Between fiscal 1998 and fiscal 2003, nearly $1.4 billion has been appropriated for the 
program. Funds have supported over 5,000 awards. The federal program requires a minimum annual 
award of $50,000; in fiscal 2003, awards averaged $98,353 annually per school. 

Congressionally mandated report. This report fulfills the evaluation requirements of the 1998 
legislation. It is a follow-up to the report that the Department released in 2000. 



Rationale for the Comprehensive School Reform Program 

Research suggests that aligning organizational structure and practices with common 
goals produces an effective organization or school. 1 Evidence from a national study of Chapter 1 
(now Title I) indicated that schools with well-integrated, coordinated approaches to teaching and 
learning had higher student achievement than schools using more targeted services. 2 Some 
evidence suggests that a schoolwide approach to school refonn may be more effective in 
producing gains in student achievement than separate initiatives that target individual aspects of 
a school. 3 Findings such as these were used to support the concept of comprehensive school 
reform. 



In the 1980s, researchers and practitioners began developing best-practice models for a 
school seeking to implement schoolwide improvements. 4 Throughout the 1990s, thousands of 
schools across the nation adopted various refonn methods. To assist schools with the cost of 
implementing these reform strategies, Congress created the Comprehensive School Reform 
Demonstration (CSRD) program in 1998 through P.L. 105-78. This legislation required all 
schools that receive CSRD grants to use the funding to adopt or develop research-based 
comprehensive school reform approaches. 

The 1998 legislation laid out nine components of comprehensive school refonn to guide 
this reform process (Exhibit 1-1). These components echo both the recommendations of the 
effective schools literature as well as the guidelines for Title I schoolwide programs. The 
overarching goal of the federal CSRD program was to help schools transform their operations so 
that each student will meet challenging state content and student performance standards. 
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Exhibit 1-1 

Eleven Components of Comprehensive School Reform 
Described in the No Child Left Behind Act 



■ Proven methods and strategies for student learning, teaching, and school management that are based on 
scientifically based research and effective practices and that have been replicated successfully in schools with 
diverse characteristics. 

■ Comprehensive design for effective school functioning, integrating instruction, assessment, classroom 
management, and professional development and aligning these functions into a schoolwide reform plan 
designed to enable all students to meet challenging state content and performance standards and address needs 
identified through a school needs assessment. 

■ Professional development. High-quality and continuous teacher and staff professional development and 
training. 

■ Measurable goals for student performance and benchmarks for meeting those goals. 

■ Support from staff. Support from school faculty, administrators, and staff. 

■ Support for staff. Support for school faculty, administrators, and staff. (Added in 2001) 

■ Parent and community involvement. Meaningful involvement of parents and the local community in 
planning and implementing school improvement activities. 

■ External assistance. High-quality external support and assistance from a comprehensive school reform entity 
(which may be a university) with experience in schoolwide reform and improvement. 

■ Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

■ Coordination of resources. Identification of how other available resources (federal, state, local, or private) 
will help the school coordinate services to support and sustain the school reform. 

■ Scientifically based research. Scientifically based research to significantly improve the academic achievement 
of students participating in such programs as compared with students in schools who have not participated in 
such programs. This requirement may also be met by strong evidence that such programs will significantly 
improve the academic achievement of participating children. (Added in 2001) 

Source: Elementary and Secondary Education Act as amended by the No Child Left Behind Act of 2001, Title I, 

Part F, Section 1606. 



The original nine components guided program implementation from 1998 through 2001. 
The 2001 reauthorization of ESEA changed the program’s name to the Comprehensive School 
Reform program (removing the word “demonstration”) and added two additional components 
(support for school staff and use of scientifically based research). Throughout this report, 
descriptions of the evaluation findings, which relate to the program as created in 1998, will refer 
to the “CSRD” program, while descriptions of the current program authorized in the 2001 
legislation will refer to the “CSR” program. The findings and discussions in this report are 
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framed by the original nine components; future implementation and research will be guided by 
the 1 1 components included in the No Child Left Behind Act of 2001. 

Between fiscal 1998 and fiscal 2003, appropriations for the program totaled nearly 
$1.4 billion (Exhibit 1-2). The majority of the money is committed to Title I eligible schools, but 
a portion of the funds are available to all public schools under the Fund for the Improvement of 
Education (FIE) program. Between 1998 and 2003, three-year grants have been provided to over 
5,000 recipients to support the initial implementation costs of adopting a research-based reform 
strategy. There is a minimum annual award of $50,000, but actual grants often exceed that 
amount; for fiscal 2003, states reported making grants that averaged $98,353 annually per 
school. 



Exhibit 1-2 

CSRD and CSR Appropriations, Number of Grantees, and Average Grant Size 





ADDrouriations ($ in millions) 


Number of Grantees 








Fund for the 




New 


Continuation 


Average 




Title I 


Improvement of 
Education (FIE) 


Total 


Awards 


Grants 


Grant Size 


FY 1998 


$120 


$25 


$145 


445 


0 


$66,669 


FY 1999 


$120 


$25 


$145 


1,389 


470 


$70,983 


FY 2000 


$170 


$50 


$220 


283 


1,869 


$70,339 


FY 2001 


$210 


$50 


$260 


1,184 


1,525 


$90,351 


FY 2002 


$235 


$75 


$310 


1,334 


1,476 


$101,322 


FY 2003 


$233 


$75 


$308 


471* 


1,422* 


$98,353* 


Total 


$1,088 


$300 


$1,388 


5,106 







Sources: U.S. Department of Education, Budget Service (appropriations figures); Southwest Educational 
Development Laboratory (SEDL) Database, May 2004 (number of grantees and average grant size). 

Notes: SEDL data on continuation grants do not match exactly the number of new awards made in previous years; 
SEDL compiled these data based on state reports, which may contain errors. Award data for 2003 are incomplete. 



Challenges to Whole School Reform 

Implementation difficulties. While a school reform model can be a vehicle for organizing 
comprehensive school reform, studies have shown that whole school change can be difficult to 
implement. Even with the external support of a model developer, various studies have shown 
that it can take several years for the reform efforts to improve student outcomes.^ Initial studies 
in the early 1990s indicated that full implementation of some reform methods was associated 
with improved student achievement but that few schools have achieved full implementation. 6 
In addition, the level and quality of implementation appears to vary widely — both within schools 
and between schools. Studies show that implementation may take hold in a few classrooms but 
not spread to all facets of a school. 7 A recent external evaluation of New American Schools 
(NAS) models found that schools with two or more challenges (e.g., internal tensions, leadership 

turnover, forced adoptions of designs, poor understanding of designs) consistently ranked low on 

8 

the outside evaluator’s implementation scale. 
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Misalignment of model programs and reform goals. Emerging research indicates that 
successful implementation of reform intervention does not guarantee improved student 
achievement . 9 Several factors may contribute to the lack of significant increases in test scores. 
Few of these interventions are comprehensive (e.g., they do not actually refonn all aspects of a 
school). In addition, nationally available model programs may not be well-aligned with the 
reforming school’s district or state accountability system. Also, a number of these models 
support general school management and organizational improvements and do not focus on 
strengthening content and pedagogy. While a best-practice model can be a powerful tool for 
school improvement, the model is but one piece of a reform effort that should help a school 
remain accountable to district and state requirements. 

Need for external assistance. External assistance may be crucial to the refonn process by 
providing schools with effective strategies and resources to improve their teaching and learning. 
Presumably, schools that are persistently low-performing would have turned around if they knew 
how. Assistance may come from a model developer, a local university, the school district, the 
state, or from federal resources. 

Department of Education Activities 

Supported with funds appropriated under both Title I and the Fund for the Improvement 
of Education (FIE), the federal CSR program is both an extension of the Title I program and a 
strategy to encourage the implementation of research-based, innovative practices in schools 
throughout the country. Accordingly, the CSR initiative includes several parts: grants to 
schools, technical assistance, and research and development. 

The largest portion is the grant program under which funding is allocated to states on a 
formula basis, with states making competitive grants to districts on behalf of individual schools. 
The federal program office administers the grant program and provides technical assistance to 
state coordinators to help them design and implement their competitive grant programs. 

To support reforms at the school level, the U.S. Department of Education has funded 
the National Clearinghouse on Comprehensive School Refonn (NCCSR), which collects and 
disseminates information on implementation and evaluation of CSR . 10 The Department has also 
contracted with regional education laboratories so that each lab can provide technical assistance 
for comprehensive school reform efforts in their region. Finally, the Department has a contract 
with the Southwest Educational Development Laboratory (SEDL) to collect data from states on 
their CSR grantees. The database is publicly available online 1 1 and provides the sampling frame 
for the national evaluation of the CSRD program. 

The Department’s Institute of Education Sciences (formerly the Office of Educational 
Research and Improvement) funds research and development of externally designed reform 
models through grants to study the effectiveness of comprehensive school reform and grants to 
model developers to expand their programs or make them more comprehensive. 
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National Evaluation of the Federal CSRD Program 

The 1998 legislation also mandated national evaluation activities, requiring the 
Department of Education “to assess results achieved by the implementation of comprehensive 
school reform in Title I schools.” The Department released its first report in 2000, describing the 
early implementation of the CSRD program. “ While the 2000 report to Congress was too early 
to provide data on student achievement outcomes, this evaluation report includes an assessment 
of CSRD at the state, district, and school levels, and provides preliminary infonnation on how 
student achievement has changed in schools receiving CSRD funds. 

Guided by the components included in the authorizing legislation, the National 
Evaluation of the CSRD Program examines the following research questions: 

• Are CSRD funds well targeted to schools with the greatest need? 

• Have schools implemented the nine components of CSRD? 

• Has student achievement improved in funded schools? 

Four primary data sources contribute to this final report: 1) the National Longitudinal 
Survey of Schools (NLSS); 2) the National School-Level State Assessment Score Database; 

3) the Field-Focused Study of CSRD Schools; and 4) the CSRD grantee database compiled by 
the Southwest Educational Development Laboratory (SEDL database). Data presented in this 
report are for various years between 1998-99 and 2001-02. 

National Longitudinal Survey of Schools. The NLSS survey, conducted by Westat with 
assistance from RAND, included longitudinal surveys of principals and teachers in nationally 
representative samples of Title I schools and CSRD schools. The surveys asked questions about 
the implementation of content and performance standards and student assessments; adequate 
yearly progress and schools identified as in need of improvement; school and family 
partnerships; teacher professional development; and the use of model programs. 

Data were collected during three consecutive school years: 1998-99, 1999-2000, and 
2000-0 1 . In the first year of data collection, the sample included 311 schools from the group of 
early recipients of CSRD awards, but the sample was expanded for the second and third years of 
the study to include an additional 721 grantees. Most NLSS data presented in this report are for 
1999-2000 and 2000-01 and are based on the full sample of 1,032 CSRD schools, but some data 
are for 1998-99 and are based on the original sample of 3 1 1 schools. The principal and up to six 
teachers in each school were surveyed using computer-assisted telephone interviewing (CATI). 
Response rates were above 80 percent for both principals and teachers in all three years. 

National School-Level State Assessment Score Database. A database constructed by the 
American Institutes of Research (AIR), under a contract with Policy and Program Studies 
Service of the U.S. Department of Education, provides data on aggregate grade-level test scores 
for each school within a state for which school-level test-score data were reported. These data 
allow comparisons of the improvement in student outcomes in CSRD and non-CSRD schools 
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within each state. This analysis includes 37 states plus the District of Columbia, which all had 
consistent school-level assessment data for at least two or three consecutive years. For nine 
states, there were three years of state assessment data, from 1998-99 to 2000-01; for 29 states, 
there were only two years of data, from 1998-99 to 1999-2000. 13 

Field-Focused Study of the CSRD Program. The Field-Focused Study, conducted by 
COSMOS Corporation with assistance from the McKenzie Group, included case studies of 18 
CSRD schools in nine districts using site visits; classroom observations; interviews with 
principals, teachers, and district CSRD coordinators; and analysis of school documents. The 
study drew a purposive sample to represent various geographic regions and the most commonly 
implemented reform interventions. Four visits were made to each school, two in the fall and 
spring of the 2000-01 school year and the other two in the fall and spring of 2001-02. 

SEDL database. In order to track the amount of CSRD funds awarded to individual 
schools, the U.S. Department of Education contracted with the Southwest Educational 
Development Laboratory (SEDL) to compile a database of grantees. As states award their 
grants, they are expected to provide the following details to SEDL: name of grantee school, 
contact information, award amount, method(s) implemented, the school’s poverty rate, whether 
the grantee is a Title I school, whether the school is a Title I schoolwide or a targeted assistance 
program, and whether the school has been identified as in need of improvement. 
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Section II: Implementation of the CSRD Program 



Highlights 

CSRD resources are targeted to high-need schools. The CSRD program is meeting the legislative 
mandate to serve high-need schools in diverse settings. Nearly half (46 percent) of all CSRD schools 
have at least three-fourths of their students eligible for free or reduced-price lunches, compared with 17 
percent of all schools. CSRD schools are also more likely to have high concentrations of minority and 
limited English proficient students. Compared with non-CSRD schools, CSRD schools are four times as 
likely to have been identified as in need of improvement under Title I. 

CSRD schools were twice as likely to adopt a reform method as Title I schoolwide programs. In the 

1999-2000 school year, every CSRD school had adopted a reform method. In contrast, only 71 percent of 
non-CSRD schools with schoolwide programs had adopted an externally-developed method. 

Staff in CSRD schools showed greater support for the school’s chosen reform method, and principals in 
CSRD schools were more likely to report that professional development activities in their school were 
influenced by school plans, student assessment data, and state or local content standards. 

In a number of other areas, CSRD schools and non-CSRD schools did not differ significantly, 
although this was often because non-CSRD schools as well as CSRD schools were highly likely to 
report practices associated with comprehensive school reform. 



Targeting of CSRD Grants 

The CSRD legislation encouraged states to award grants to schools and districts to 
support high-poverty, under-perfonning schools in diverse settings. Programs like CSRD and 
Title I offer additional federal support to high-need schools to help increase the educational 
opportunities for high-need students. CSRD schools are more likely to have high concentrations 
of minority students and students from low-income families. CSRD schools are also more likely 
to be identified as in need of improvement under Title I. 

School poverty level. Nearly half (46 percent) of CSRD schools were in the highest 
poverty category (defined as having more than 75 percent of their students eligible for free or 
reduced-price lunches), compared with only 17 percent of all schools in the U.S. 

Even among schools with Title I schoolwide programs, which usually have poverty rates 
above 50 percent (unless a waiver has been granted), CSRD schoolwides were much more likely 
to be in the highest poverty group (56 percent) than were non-CSRD schoolwides (35 percent). 
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Schools in need of improvement. States tend to award CSRD grants to schools that 
have been identified under Title I as in need of improvement: 37 percent of CSRD schools have 
been identified as in need of improvement, compared with 17 percent of Title I schools. Title I 
schools that have been identified for improvement account for approximately 9 percent of all 
schools. 

Minority students. CSRD schools were much more likely to serve high proportions of 
students from racial or ethnic minorities. Almost half (46 percent) of all CSRD schools, and 
53 percent of CSRD schoolwides, served student populations that were over 75 percent minority. 
This degree of racial isolation was present in only 28 percent of Title I schoolwide schools and 
16 percent of all schools nationwide. 

LEP and migrant students. CSRD schools also tended to serve higher concentrations 
of limited English proficient (LEP) students and migrant students. More than one-fourth (29 
percent) of CSRD schools serve student populations that are over 10 percent LEP, compared 
with only 15 percent of all schools. Similarly, 26 percent of CSRD schools serve migrant 
students, compared with 18 percent of all schools. 

Urbanicity. CSRD schools were almost twice as likely as the average school to be 
located in an urban area (44 percent of CSRD schools, compared with 24 percent of all schools). 
CSRD schools were much less likely to be located in suburban areas and large towns (30 percent 
vs. 45 percent) and slightly less likely to be located in rural areas (26 percent vs. 31 percent). 

Grade level. Elementary schools were much more likely to receive CSRD grants than 
middle schools or high schools. In 2000-01, 71 percent of all CSRD schools were elementary 
schools, compared with 59 percent of all schools nationally. High schools accounted for 1 1 
percent of CSRD schools and 22 percent of all schools, while middle and junior high schools 
were equally represented among CSRD schools and all schools (15 percent). 
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Exhibit II- 1 

Demographic Characteristics of CSRD Schools Compared with All Schools 
and CSRD Schoolwides Compared with Non-CSRD Schoolwides 





CSRD 

Schools 


All 

Schools 


CSRD 

Schoolwide 

Schools 


Non-CSRD 

Schoolwide 

Schools 


Percent of students eligible for free/reduced price lunch 










75-100% 


46%* 


17% 


56%* 


35% 


50-74.9% 


3 1%* 


19% 


32% 


41% 


35-49.9% 


9%* 


16% 


5%* 


18% 


0-34.9% 


14%* 


47% 


7% 


6% 


Percent of schools identified as in need of improvement 


37%* 


9% 


41%* 


20% 


under Title I 










Percent of minority students 










75-100% 


46%* 


16% 


53%* 


28% 


50-74.9% 


18%* 


12% 


19% 


16% 


25-49.9% 


14% 


16% 


13%* 


23% 


0-24.9% 


22% * 


55% 


15%* 


33% 


Percent of students with limited English proficiency (LEP) 










25-100% 


15%* 


7% 


17% 


16% 


10-24.9% 


14% * 


8% 


14%* 


9% 


0. 1-9.9% 


27%* 


39% 


25% 


26% 


0% 


44% 


46% 


43% 


48% 


Number of migrant students 










50 or more migrant students 


6% 


3% 


6% 


6% 


1 -49 migrant students 


20%* 


15% 


21% 


29% 


None 


74% * 


82% 


74% 


65% 


Urbanicity 










Urban 


44% * 


24% 


48%* 


31% 


Suburban/Large Town 


30%* 


45% 


28% 


28% 


Rural/Small Town 


26%* 


31% 


25%* 


41% 


School grade level 










Elementary schools 


71%* 


59% 


79% 


84% 


Middle/junior high schools 


15% 


15% 


13% 


9% 


High schools 


1 1%* 


22% 


5% 


5% 


Other 


3% 


6% 


3% 


3% 



Exhibit reads: The highest-poverty group of schools, in which 75 percent or more of the students are eligible 
for free or reduced-price lunches, accounted for 46 percent of all CSRD schools, 17 percent of all schools, 

56 percent of CSRD schools with schoolwide programs, and 35 percent of non-CSRD Title I schoolwides. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 



Sources: NLSS Principal Survey, 1999-2000, and Schools and Staffing Survey, 1999-2000. Sample sizes for 
responding principals in the NLSS were 837 CSRD schools, 652 CSRD schoolwides, and 588 non-CSRD 
schoolwides. The sample size for the estimates for all schools from the SASS were 9,302 schools. 



Note: The percent of all schools identified as in need of improvement under Title I was calculated by dividing the 
total number of identified schools reported in State Consolidated Performance Reports (8,505) by the total number 
of schools reported by the Common Core of Data (92,012) in 1999-2000. 
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Implementing the Components of the Federal Program 

During the time period covered by the studies included in this report, grantees were 
expected to implement the nine components of the 1998 CSRD legislation (Exhibit II-2). 



Exhibit II-2 

Nine Components of the CSRD Program 

■ Innovative strategies and proven methods that are based on reliable research and effective practices and that 
have been replicated in schools with diverse characteristics. 

■ Comprehensive design for effective school functioning, integrating all facets of teaching and learning into a 
schoolwide reform plan to enable all students to meet challenging state content and performance. 

■ High-quality and continuous teacher and staff professional development and training. 

■ Measurable goals for student performance and benchmarks for meeting those goals. 

■ Support from school faculty, administrators, and staff. 

■ Involvement of parents and the local community in school improvement activities. 

■ External support and assistance from an organization with experience in schoolwide reform and improvement. 

■ Plan to evaluate the implementation of school reforms and the student results achieved. 

■ Coordination of resources and services (federal, state, local, or private) to support and sustain the school 
reform. 

Source: Public Law 105-78, the Fiscal Year 1998 Appropriations Act for the U.S. Department of Education. 



The main focus of the CSRD program is on implementing innovative strategies and 
proven methods as part of a plan for whole-school change. These strategies are generally 
introduced through professional development, often delivered through external technical 
assistance. Examining the implementation of these three components provides the best sense of 
how the CSRD program has been implemented. For that reason, this section will focus on these 
components. There is less infonnation on other components, but they also provide insight into 
the extent and the quality of program implementation. 

Innovative strategies and proven methods. The CSRD program was intended to 
promote innovative strategies and proven methods for student learning and teaching, which often 
has often been interpreted as implementing an externally developed reform method or model. 

The reform can be externally developed and available nationally or regionally, or the school can 
adapt its own program drawing from the research literature. According to the NLSS data, by 
2000-01, every CSRD school had adopted an externally developed reform method, compared 
with 71 percent of non-CSRD schoolwide schools. Findings from the Field-Focused Study 
indicate that the implementation process is similar regardless of whether the reform method is 
national or locally developed: in both cases, schools receive external technical assistance and 
make similar progress toward implementing the program components. 
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As of February 2004, CSRD and CSR schools had implemented 875 different reform 
methods. More than one-third (35 percent) of grantees had chosen a method listed in the original 
CSRD legislation. More than half (5 1 percent) had chosen one listed in the Northwest Regional 
Educational Lab catalog (NWREL). 14 The most commonly implemented method was Success 
for All, selected by 9 percent of grantees. The 10 most common methods account for 38 percent 
of all grantees. 



Exhibit II-3 

Ten Most Common Methods Implemented by CSR Schools 





Number of 
CSR Schools 


Percent of 
CSR Schools 


Success for All 


446 


9% 


Lightspan 


243 


5% 


Accelerated Schools 


230 


5% 


America's Choice 


184 


4% 


Co-NECT 


165 


3% 


Coalition of Essential Schools 


162 


3% 


High Schools that Work 


138 


3% 


Direct Instruction 


137 


3% 


Effective Schools 


135 


3% 


School Development Program 


112 


2% 


Total 


1,952 


38% 



Source: SEDL Database, February 2004. 



Note: Schools in this table include both those that received grants under the 
original CSRD program as well as the subsequent CSR program. 



However, case study findings from the Field Focused Study indicate that implementation 
of the chosen research-based method was uneven across the 18 sites. Half of the CSRD schools 
were fully implementing their chosen method; elements of the method were observed in all or 
nearly all classrooms in these nine schools, and faculty appeared to be comfortable with using 
the method. Another one-third of the schools were implementing some components of the 
method, but this implementation was uneven or faltering. In some of these schools, use of the 
method was inconsistent across teachers or implemented at only one grade level. Other schools 
were implementing portions of the method schoolwide but had not yet implemented all 
components of the method. Finally, at three of the schools, implementation was minimal or had 
stalled completely. 



The models listed in the 1998 legislation included: Accelerated Schools, ATLAS Communities, Audrey Cohen 
College, Coalition of Essential Schools, Community for Learning, Comer School Development Program, Co-NECT, 
Direct Instruction, Expeditionary Learning Outward Bound, High Schools that Work, Modern Red Schoolhouse, 
Paideia, Roots and Wings, Success for All, Talent Development High School, and Urban Learning Centers. 
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In addition, fidelity to the original method was also uneven, as judged by the model 
developers. In the 14 schools for which the study was able to obtain judgments about fidelity of 
implementation from the model developer, the developer judged fidelity to be high in six of the 
schools, moderate in six schools, and low in two of the schools. 

How Schools Chose Reform Methods. School decisions to adopt a particular refonn 
method are often not completely voluntary decisions. According to the NLSS, in 2000-01, 
almost two-thirds (63 percent) of CSRD schoolwide schools had adopted their primary reform 
method either as a result of being identified as in need of improvement or at the direction of the 
district or the community, compared with 53 percent of non-CSRD schoolwide schools. Being 
identified for improvement was the reason for adopting a reform method in 44 percent of CSRD 
schoolwides but only 16 percent of non-CSRD schoolwides. 

Virtually all CSRD schools in the NLSS sample reported that they participated in a 
careful selection process, the first step of which generally consisted of a thorough needs 
assessment (this was also true for non-CSRD schoolwides). In making decisions to adopt a 
method or methods, 96 percent of CSRD principals said that research evidence was an important 
factor in their selection; 95 percent of principals of non-CSRD schoolwides also gave this 
response. The overwhelming majority of CSRD schoolwide principals reported that they had 
sufficient access to information on the various methods, sufficient information about the needs of 
their own schools, and the time needed to explore various options. (This was also true for non- 
CSRD schoolwides.) 

Despite these findings, only 40 percent of CSR grantees have used a method rated as 
having “strong” or “promising” evidence of effectiveness by the Educators ’ Guide to Schoolwide 
Reform ; 26 percent have used a method rated as “strong.” 15 Although principals appear eager to 
make informed choices about refonn programs, the fact remains that strong evidence on the 
effectiveness of many models is often not available. 

Schools may adopt more than one externally developed refonn method, and the NLSS 
data show that 21 percent of CSRD grantees had adopted two methods. In contrast, 43 percent 
of non-CSRD schoolwides had adopted one reform method and 3 1 percent had adopted two. The 
Field-Focused Study also found that some reform methods are being deliberately coupled with 
other methods. Since research has shown few of the nationally available methods are 
comprehensive, adopting more than one method may be a strategy to reach all aspects of a 
school. 



Implementation Strategies. Analysis of implementation data from the Field-Focused 
Study suggests three pathways that lead to reform. First, a school can explicitly implement each 
of the nine components. Second, a school can adopt a research-based method that requires 
comprehensive changes in school operations. A comprehensive method can help a school 
implement the other eight CSRD components, without necessarily addressing each component 
individually. Third, a school can be part of a school system that fosters whole school change 
because state, district, and school policies and practices are aligned toward school reform. 
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While each path can lead to sustainable reforms, the implementation process looks 
different in each type of school. Some schools write a school refonn plan that explicitly 
incorporates the program components. This written plan provides a blueprint for school refonn. 
By implementing the program components, within a plan for change, schools meet the program 
requirements. Other schools implement a comprehensive, externally designed refonn method. 
The method, coupled with external technical assistance, can lead CSRD grantees to whole-school 
change. 



In other cases, a strong, well-aligned accountability system can provide continuing 
support for reforms. Because the effort is led by the district or state, it incorporates local and 
state expectations. Unfortunately, if the district or state modifies policy and adopts a different 
reform method than the one included in the CSRD plan, schools can be forced to radically 
change strategies and may lose momentum. In other instances, schools are implementing 
comprehensive school reform because they have been identified by the district or state as 
underperforming or in need of improvement. This identification may result in the school 
receiving additional assistance and financial support from the state or district. If these resources 
are aligned with the CSRD-funded program, the combined effort could lead to more sustainable 
implementation of the reforms. 

Barriers to Implementation. The majority of principals believe that they are able to 
implement their reform methods without much difficulty. Few principals (3 percent of CSRD 
schoolwides and 2 percent of non-CSRD schoolwides) reported that they were finding it difficult 
“to a great extent” to implement the reform. Slightly less than one-quarter of principals in CSRD 
schoolwide schools and non-CSRD schoolwide schools with refonn methods reported that they 
were finding it moderately difficult to implement the method. This relatively sanguine view 
contrasts with the findings of the Field-Focused Study that only half of the 18 CSRD schools 
visited had fully implemented their chosen method and that the fidelity of implementation, as 
judged by the model developers, was considered “high” in less than half of these schools. 

Among those principals experiencing implementation difficulties, the two greatest 
problems reported were insufficient planning time and staff turnover. Inadequate funding and 
resources were reported to be more of a barrier for non-CSRD schoolwide schools with reform 
methods (23 percent) than for CSRD schoolwide schools (14 percent), suggesting that CSRD 
grants may help reduce financial barriers to implementing reform. 

Professional development. Professional development is an important component of the 
CSRD program, providing assistance to enable educators to improve teaching and learning for 
students in their schools. Teachers in CSRD and non-CSRD schoolwides were about equally 
likely to report receiving professional development (four-fifths of teachers in both groups), but 
teachers in CSRD schools reported fewer total hours of professional development than teachers 
in non-CSRD schoolwide schools. Findings varied by school level; elementary school teachers 
in CSRD schools were more likely to obtain professional development in reading and math than 
their counterparts in non-CSRD schools, but no significant difference was found for secondary 
school teachers. Principals in CSRD schools were more likely to report that professional 
development activities in their school were influenced by school plans, student assessment data, 
and state or local content standards. 
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Exhibit II-4 

Professional Development in CSRD and Non-CSRD Schoolwide Schools 





CSRD 


Non-CSRD 




Schools 


Schools 


Percent of teachers who participated in professional development in the past year 




Professional development in their content area 


82% 


79% 


Average number of PD hours in content area 


28 hours* 


34 hours 


Professional development in reading 






Elementary teachers 


88%* 


82% 


Secondary teachers 


80% 


80% 


Professional development in math 






Elementary teachers 


68%* 


61% 


Secondary teachers 


85% 


87% 


Other tvpes of professional development 






Common planning time 


97% 


98% 


Formal mentoring relationships 


57%* 


53% 


Networking with outside teachers 


66% 


67% 


Important influences on professional development, as reported by principals 




School plans 


87%* 


72% 


Student assessment data 


78%* 


65% 


Implementation of content standards 


74%* 


62% 



Exhibit reads: Although teachers in both CSRD and non-CSRD schoolwides were equally likely to report 
receiving professional development in their content area, teachers in CSRD schoolwides reported receiving 
fewer hours (28) of such professional development than teachers in non-CSRD schoolwides (34). 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 

Source: NLSS Teacher and Principal Surveys, SY 1999-2000 and 2000-01. The sample size for responding 
principals ranged from 651 to 676 in CSRD schoolwides and from 588 to 611 in non-CSRD schoolwides, 
depending on the year of the survey. For teachers, the sample size ranged from 3,571 to 3,577 in CSRD 
schoolwides and from 3,183 to 3,301 in non-CSRD schoolwides 



The Field-Focused Study found that nearly all of the case study schools were engaged in 
moderate to intensive professional development, and most of this professional development was 
provided through an external technical support team. The professional development was largely 
focused on implementation of a model program, not necessarily comprehensive reform more 
broadly. 

External technical assistance. According to the NLSS data, 97 percent of CSRD schoolwide 
schools that were beyond the initial selection and planning phase had received assistance in 
implementing the method during the past year. (Comparable infonnation for non-CSRD schools is 
not available.) The Field-Focused Study found no differences in the amount or type of technical 
assistance received by schools using nationally or locally developed methods, because the locally 
developed methods usually had a regional or university-based developer that assisted the school. 

Much of the technical assistance was provided by the developer of the school’s refonn model. 
Fifty-six percent of principals in CSRD schoolwide schools reported that the model developer was the 
primary provider of professional development or technical assistance in implementing the model. 
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This technical assistance focused on implementing the external model and not necessarily on reforms 
aligned to the school’s accountability requirements. 

CSRD schools identified as in need of improvement were more likely than other CSRD 
schools to receive additional technical assistance from a variety of sources. These schools receive 
more assistance from school districts, state departments of education, school support teams, university 
staffs, and independent consultants. 

Comprehensive plan for school reform. NLSS data indicate that almost all schoolwide 
schools (both CSRD and non-CSRD) had an overall written annual or strategic plan in 2000-01. 
However, the Field-Focused Study found that less than half of the case study schools (eight out 
of 18) had in place reforms that were truly comprehensive. Six of the 18 schools had 
comprehensive plans or documentation, such as school improvement plans, but staff were 
unaware of or were detached from any comprehensive vision associated with these plans. 

Among most of the remaining schools, there was neither documentation nor staff understanding 
that demonstrated that the school had a comprehensive plan for improving the school. 

Technical assistance typically did not address the comprehensiveness of a school’s 
reform plan. The Field-Focused Study found that external technical assistance largely focused 
on a specific reform method. However, this assistance did not address the comprehensive, 
whole-school design of the reform unless the method itself was comprehensive. Many model 
programs are by design not comprehensive because they are limited to certain grades or certain 
academic subjects. 

Measurable goals and benchmarks. The NLSS study examined several indicators of 
whether schools had measurable goals and benchmarks and found no differences between CSRD 
and comparable non-CSRD schools. Nearly all CSRD schoolwide schools reported having 
quantifiable goals for how student achievement should improve every year (96 percent in 
2000-01) and had adopted reform methods that included a student assessment component as 
well as goals and benchmarks for students (90 percent in 1999-2000). The Field-Focused Study 
noted that goals and benchmarks were usually focused on student achievement and typically did 
not measure progress in implementing the nine CSRD components. 

Support from school staff. Teachers in CSRD schoolwide schools were more positive 
about the effects of the method on their teaching and their students, with between 40 to 50 
percent reporting that it had improved their teaching, professional growth, and students’ 
engagement in learning “to a great extent,” compared with 33 to 41 percent of teachers in non- 
CSRD schoolwide schools. However, there was no significant difference between CSRD and 
comparable schools in the percent of teachers reporting that they supported the reform method 
“to a great extent.” The Field-Focused Study found that high turnover among staff at some 
schools can result in a transient level of support for the reform effort. In particular, turnover 
among principals can sway support or involvement when new principals have agendas and 
priorities that differ from their predecessors’ agendas, since strong principal leadership can be a 
critical ingredient in the success of a reform effort. 
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Exhibit II-5 

Implementation of Selected CSRD Program Components 
at CSRD and Non-CSRD Schoolwide Schools 



CSRD Non-CSRD 

Schools Schools 

Innovative strategies and methods 

Adoption of externally developed reform method 100%* 71% 

Comprehensive plan for school reform 

Overall written annual or strategic plan in SY 2000-01 99% 99% 

Measurable goals and benchmarks 

Quantifiable annual goals for student achievement 96% 89% 

Adopted reform methods with student assessments, goals, 90% 93% 

and benchmarks 

Written plans to improve student achievement (reading) 96% 87% 

Written plans to improve student achievement (math) 85% 84% 

Support from school staff 

Teachers report supporting reform method to “great extent” 61% 55% 

Teachers report that reform method improved their teaching “to a 40%* 33% 

great extent” 

Teachers report that reform method improved their professional 44%* 36% 

growth “to a great extent” 

Teachers report that reform method improved students’ engagement 50%* 41% 

in learning “to a great extent” 

Parent involvement 

Principals report use of school-parent compacts 90% 84% 

Teachers report use of school-parent compacts 64% 64% 

Teachers discussed performance goals with parents 82% 81% 

Teachers discussed how teachers and parents could help 98% 99% 

the student achieve these goals 

Evaluation 

Primary reform method included a student assessment component 95% 98% 

Coordination of Resources 

Combined their federal funds with other funding sources 52% 44% 

“to a great extent” 

Principals uncertain about what is allowed 31%* 41% 



Exhibit reads: Nearly all (96 percent) CSRD schoolwide schools reported having quantifiable annual 
goals for student achievement. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 

Source: NLSS Teacher and Principal Surveys, 1998-99, 1999-2000, and 2000-01. The sample size for 
responding principals ranged from 245 to 676 in CSRD schoolwides and from 588 to 611 in non-CSRD 
schoolwides, depending on the year of the survey. For teachers, the sample size ranged from 1,069 to 3,577 
in CSRD schoolwides and from 2,822 to 3,301 in non-CSRD schoolwides 
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Parent and community involvement. No significant differences were found between 
CSRD and comparable schools in principal and teacher reports on parent involvement. Almost 
all CSRD schools reported that parents played a role in developing the strategic plan. Most 
principals reported using school-parent compacts that spell out the responsibilities of the school 
and the parents (90 percent of CSRD schoolwides). Although teachers in these schools were less 
likely than their principals to report the use of compacts (64 percent), they usually reported 
talking with parents about student performance goals (82 percent) and about how the teacher and 
parents could help the student achieve these goals (98 percent). 

The Field-Focused Study found that the extent of parent involvement can be transient, 
particularly where there is high student turnover, and high involvement during the kickoff year 
of a reform effort may not have been sustained the following year. The study found that few 
schools had defined strategies to produce broad and sustained parental and community 
involvement; in most cases the observed support and involvement was based on the use of CSRD 
funds to hire a parent coordinator or isolated milestones such as the convening of a parents’ 
night. 



Evaluation. The Field-Focused Study found that schools do not necessarily have explicit 
evaluation plans. However, evidence from the NLSS indicates that CSRD grantees may be 
relying on their externally developed reform method’s evaluation process. In schools adopting 
reform methods — both CSRD and non-CSRD — well over 90 percent reported that their primary 
reform method included a student assessment component. 

Coordination of Resources. Because the CSRD grant is provided for only three years, 
it is important for grantees to coordinate and leverage other funding sources in order to continue 
the reforms beyond the duration of the federal grant. In SY 1998-99, half (52 percent) of the 
principals of CSRD schoolwide schools reported combining their federal funds with other 
funding sources “to a great extent,” compared with 44 percent of principals of non-CSRD 
schoolwides. Among CSRD grantees, elementary schools were significantly more likely to 
integrate funds than were secondary schools. Both CSRD and non-CSRD schoolwide principals 
reported that district and state control over the use of funds was a challenge for about half the 
CSRD schoolwide principals in the NLSS sample. However, CSRD schoolwide principals were 
less likely to report uncertainty about what is allowed as a barrier to the coordination of 
resources (31 percent) than principals in non-CSRD schoolwide schools (41 percent). 

The Field-Focused Study found that while most of the case study schools were 
coordinating funds to support the reform effort, some schools were coordinating only funds from 
federal sources because they did not have the access to local funds needed to redirect them. The 
study also found that nearly all of the schools were unclear about the amount or source of new 
funds that might be used to implement the research-based method after the CSRD award has 
ended. 



17 




18 




Section III: Achievement Trends for CSRD Schools 



Highlights 

This analysis of student outcomes should be considered preliminary because the time frame covered 
here is too short to expect large measurable effects of the CSRD program 

Most schools included in this analysis had been receiving CSRD funding for one or two years, and the 
state assessment data that was available provided only one or two years of achievement change data. 

This early evidence found that some CSRD schools made gains on state assessments, but no 
relationship was found between CSRD funding and student achievement. 

CSRD schools made gains in reading and mathematics in about one quarter of the states; however, 
non-CSRD schools also improved on state assessments. 

Looking at changes in the achievement ranking of CSRD schools compared with other schools in the 
state, states were more likely to show positive average gains than declines for CSRD schools, particularly 
at the elementary level. These gains were more prevalent in states that had assessment trend data 
available for three years instead of the usual two years. However, few of these relative achievement gains 
were statistically significant. 



The intent of the CSRD program is to raise student achievement by promoting 
schoolwide reform and improvement strategies. If the program is effective, we would expect to 
see achievement growth trends in CSRD schools that outpace achievement growth in other 
schools with similar demographics. This section examines trends in student achievement on 
state assessments. However, it is important to note that insufficient data are currently available 
for a meaningful examination of achievement trends in CSRD schools. According to the 
literature on school change, schools may require more than five years to accomplish significant 
schoolwide reforms, but this report is only able to examine one- or two-year trends from the 
baseline year to the first or second year after receipt of CSRD funds. 

Therefore, this report provides only a preliminary examination of achievement trends in 
CSRD schools, and further analysis is needed as additional years of state assessment data 
become available in order to understand the impact of the CSRD program on student 
achievement. 

This achievement analysis focuses on two evaluation questions; 

■ Have CSRD schools made statistically significant gains from the baseline year 
(1998-99) to the first or second year after receipt of CSRD funds? 
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Have CSRD schools made relative gains compared with non-CSRD schools within 
the state from the baseline year (1998-99) to the first or second year after receipt of 
CSRD firnds? 



Methodology 

This analysis uses data from the National School-Level State Assessment Score 
Database, 16 which provides data on aggregate school-level test scores for each grade tested on 
state assessments. The database also includes data on Title I status and school poverty levels (as 
measured by the percent of students eligible for the free and reduced-price lunch program). To 
determine which schools in this dataset received CSRD funds, it was merged with data compiled 
by the Southwest Educational Development Laboratory (SEDL). 17 

State assessments provide information on student progress toward meeting state academic 
content and academic achievement standards. However, these data cannot be aggregated across 
states to examine national trends or used to make comparisons among states. Because each state 
has developed its own standards and assessments, there is little comparability across states in the 
content and rigor of these standards and assessments. Therefore, this analysis is based on 
separate state -by-state comparisons of CSRD and non-CSRD schools within each state. 

This analysis includes 37 states plus the District of Columbia. Five states were excluded 
because they did not have consistent school-level assessment data available (Iowa, Montana, 
Nebraska, New Mexico, and North Dakota). Seven additional states were excluded because they 
had only one year of state assessment data available (Alaska, Hawaii, Kansas, Louisiana, 
Michigan, Missouri, and Nevada). Mississippi was also excluded because their state assessment 
changed during the period covered by this analysis. 



Exhibit III-l 

Metrics Used by States in Reporting School-Level Test Scores 



Test Score Metric 


# of States 


States 


Scale score or normal curve 
equivalent (NCE) score 


15 


CA, DE, FL, IL, KY, MA, ME, MN, MS, NC, NH, NJ, NY, 
PA, SC 


Percent passing level 


21 


AK, CO, CT, DC, HI, IN, KS, LA, MD, MI, MO, OH, OK, 
OR, RI, TX, VA, VT, WA, WI, WY 


Percentile 


11 


AL, AR, AZ, GA, ID, MS, NV, SD, TN, UT, WV 


No uniform, school-level 
test scores 


5 


IA, MT, ND, NE, NM 
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The available assessment data at the time of this analysis span a different period of time 
in each state. For nine states, there were three years of state assessment data, from 1998-99 to 
2000-01 (these states are California, Florida, Kentucky, Massachusetts, New York, Texas, Utah, 
Virginia, and Washington). For 29 states, there were only two years of data, from 1998-99 to 
1999-2000. 

Not only is this period of time very short for making reasonable judgments about 
achievement effects, but the amount of time that schools had to implement their CSRD grants 
may have been even shorter due to delays in some states awarding CSRD grants. For these 
reasons, the analyses presented here have limited value forjudging the effects of the CSRD 
program on student achievement. Future studies may be able to use achievement data covering a 
longer period of time, allowing for more time to fully implement CSRD reforms. In the 
meantime, the following analyses provide some preliminary information about how achievement 
trends in CSRD schools compare with those in non-CSRD schools. 

CSRD schools were compared with all non-CSRD schools in the state. Because the 
program emphasizes selecting schools identified in need of improvement, CSRD schools are 
likely to be both higher-poverty and lower-achieving, on average, than other schools in the state, 
and it might be preferable to compare CSRD schools with a comparison group of non-CSRD 
schools with similar characteristics in terms of poverty level and prior achievement. Early 
analyses conducted for this report included selection of a comparison group of non-CSRD 
schools, but there were concerns about whether the methods used to select these comparison 
schools actually resulted in a truly comparable comparison group. Because this analysis 
examines whether CSRD schools moved up or down in their state ranking, it was not necessary 
to restrict the non-CSRD samples to a comparable group of schools. Indeed, the results 
comparing CSRD schools with non-CSRD schools were slightly more positive than the 
comparison with the selected comparison group, and the analysis using all non-CSRD schools is 
the one presented in this report. 

The findings that follow are based on the average change in test scores for CSRD schools 
from the baseline year to one or two years after receipt of CSRD funds, to see whether these 
schools improved over time. For states that report percentile ranks, the analyses are more limited 
because percentile ranks cannot be averaged across schools as with test scores; thus, these states 
are not included in the first analysis, which relies on averaging test scores across schools. More 
detailed information on the methods used in these analyses is provided in Appendix A. 

It should be noted that comparisons of grade-level scores over successive years (e.g., 
comparing the fourth-grade test scores in 1999 with the fourth-grade test scores in 2001) contrast 
the achievement of different cohorts of students. These scores are likely to vary from year to 
year because different students are taking the test. In addition, average test scores in small 
schools are likely to fluctuate widely from year to year because of the small number of students 
taking the test. A more rigorous research design for examining the impact of the CSRD program 
on student achievement would follow students longitudinally to examine whether they make 
greater gains than a comparison group of students in non-CSRD schools. Nevertheless, school- 
level achievement trends on state assessments provide one indicator of program impact that 
should continue to be monitored over a longer period of time. 
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Findings 



CSRD schools made gains in reading and mathematics in about one-quarter of the states; 
most gains were not statistically significant. 

Exhibit III-2 summarizes whether average test scores for CSRD schools increased or 
decreased from the baseline year of 1998-99 to one or two years after receipt of CSRD funds. 
This analysis includes 28 states that reported test scores as scale scores, mean normal curve 
equivalent (NCE) scores, or percentage passing. States that reported test results only in terms of 
percentiles are not included in this analysis. 

Overall, CSRD schools made statistically significant gains in reading and mathematics in 
about one-quarter of the states. In most states (70 percent, on average), there was no significant 
change in student achievement in CSRD schools, while significant declines occurred in 6 percent 
of the cases. 

Significant gains were most common at the elementary level (about one -third of the 
states). In reading, CSRD elementary schools made gains in reading in 10 out of 28 states, were 
unchanged in 14 states, and declined in four states. In mathematics, CSRD elementary schools 
made gains in nine out of 27 states and did not change in 18 states; no state showed a decline in 
elementary math achievement for CSRD schools. Middle schools and high schools were less 
likely to show positive gains. At both levels, average change in student achievement for CSRD 
schools was not statistically significant for about three-quarters of the states. Among middle 
schools, CSRD showed significant gains in four out of 23 states. High school results were 
available for only 14 states. 



Exhibit III-2 

Summary of State-Level Changes in CSRD Schools 
From Baseline to One to Two Years after Receipt of CSRD Funds 





Number 
of States 


Positive 


No Change* 


Negative 


Elementary Reading 


28 


10 


14 


4 


Elementary Math 


27 


9 


18 


0 


Middle School Reading 


23 


4 


18 


1 


Middle School Math 


23 


4 


18 


1 


High School Reading 


14 


1 


11 


2 


High School Math 


14 


3 


11 


0 


Total 


129 


31 


90 


8 


% of Cases 




24% 


70% 


6% 



* Includes states where change was not statistically significant. Statistical significance was 
evaluated at the .05 level of confidence using a two-sample t-test. 



Exhibit reads: State reading assessments for elementary schools showed positive growth 
for CSRD schools in 10 out of 28 states; negative changes occurred in four states, and 
there was no significant change in 14 states. 
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The above results are dominated by states with only one year of change data, and 
comprehensive school reforms may take a longer period of time to be implemented and begin to 
have an impact on student achievement. According to the literature on school change, schools 
may require more than five years to accomplish meaningful schoolwide refonns. 

Exhibit III-3 shows the same analysis looking only at those states with two years of 
change data (that is, a baseline year plus two years after the receipt of CSRD funds). The two- 
year change data show much more positive results for CSRD schools. At the elementary level, 
CSRD schools made gains in reading and mathematics in seven out of eight states. Middle 
school and high school results were less striking but were still more likely to be positive than the 
results including states with only one year of change data. Overall, CSRD schools showed 
achievement gains in 48 percent of the cases. 



Exhibit III- 3 

Summary of State-Level Changes in CSRD Schools 
From Baseline to Two Years after Receipt of CSRD Funds 





Number — 




All States 




X l UlllUvl 

of States 


Positive 


No Change* 


Negative 


Elementary Reading 


8 


7 


1 


0 


Elementary Math 


8 


7 


1 


0 


Middle School Reading 


8 


2 


5 


1 


Middle School Math 


8 


3 


5 


0 


High School Reading 


8 


1 


6 


1 


High School Math 


8 


3 


5 


0 


Total 


48 


23 


23 


2 


% of Cases 




48% 


48% 


4% 



* Includes states where change was not statistically significant. Statistical significance was 
evaluated at the .05 level of confidence using a two-sample t-test. 



Exhibit reads: Looking only at states with three years of assessment data, state reading 
assessments for elementary schools showed positive growth for CSRD schools in seven 
out of eight states. 
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Achievement improved for both CSRD and non-CSRD schools. 

However, it may be the case that the CSRD program itself was not the cause of the 
significant growth in student achievement for CSRD schools in these states. Exhibit III-4 shows 
that those states with significant growth in achievement for CSRD schools at the elementary 
level also had significant growth in achievement for non-CSRD schools as well (both the 
comparison schools and all non-CSRD schools). Data for middle schools and high schools show 
similar patterns. 



Exhibit III-4 



States with Significant Growth in Achievement for CSRD Elementary Schools 






Reading 


Mathematics 




CSRD 


Non-CSRD 


CSRD 


Non-CSRD 




Schools 


Schools 


Schools 


Schools 


States with Two Years of Assessment Data (1998-99 to 1999-2000) 


Colorado 


+ 


+ 


NS 


NS 


Minnesota 


+ 


+ 


+ 


+ 


South Carolina 


+ 


+ 


+ 


+ 


States with Three Years of Assessment Data (1998-99 to 2000-01) 


California 


+ 


+ 


+ 


+ 


Florida 


+ 


+ 


+ 


+ 


Kentucky 


+ 


+ 


+ 


+ 


Massachusetts 


+ 


+ 


NS 


NS 


New York 


+ 


+ 


+ 


+ 


Texas 


+ 


+ 


+ 


+ 


Virginia 


NS 


NS 


+ 


+ 


Washington 


+ 


+ 


+ 


+ 



* Statistical significance was evaluated at the .05 level of confidence using a two-sample 
t-test. “NS” indicates that the change in achievement was not statistically significant. 



Exhibit reads: In Colorado, CSRD schools made significant gains on state reading 
assessments from 1998-99 to 1999-2000, but comparison schools and all non-CSRD 
schools also had significant achievement growth. 
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CSRD schools improved relative to non-CSRD schools in slightly more than half of the 
states, but few of these relative achievement gains were statistically significant. 



Exhibit III-5 summarizes state-level findings on changes in performance made by CSRD 
schools relative to all non-CSRD schools in their states. This analysis examines change in the 
relative ranking of CSRD schools within each state to determine whether CSRD schools, on 
average, moved up in this ranking. Very few states showed a statistically significant change in 
the ranking of CSRD schools relative to non-CSRD schools. Looking at all changes, regardless 
of statistical significance, states showed improvement in the ranking of CSRD schools in 55 
percent of the cases and declines in 43 percent of the cases. 

As before, looking only at states with two years of change data shows somewhat more 
positive results for CSRD schools. Among these states, CSRD schools showed relative 
achievement gains in 57 percent of the cases, and they lost ground relative to comparison schools 
in 39 percent of the cases. 

Relative gains for CSRD schools were more common at the elementary school level. At 
the middle school level, states were about equally likely to show relative gains and relative losses 
in the ranking of CSRD schools versus other schools. High school results in reading mirrored 
the middle school patterns, but math results were more likely to show gains for CSRD schools. 



Exhibit III-5 

Summary of Change in Within-State Rankings for CSRD Schools 





All States 


States with Two Years of Change Data 


Number 
of States 


Positive 


No 

Change 


Negative 


Number 
of States 


Positive 


No 

Change 


Negative 


Elementary Reading 


38 


21 


1 


16 


9 


6 


0 


3 


Elementary Math 


37 


22 


2 


13 


9 


6 


1 


2 


Middle School Reading 


29 


15 


0 


14 


8 


3 


0 


5 


Middle School Math 


29 


15 


0 


14 


8 


4 


0 


4 


High School Reading 


15 


7 


0 


8 


6 


3 


0 


3 


High School Math 


15 


9 


1 


5 


6 


4 


2 


1 


Total 


163 


89 


4 


70 


46 


26 


2 


18 


% of Cases 




55% 


2% 


43% 




57% 


4% 


39% 



Exhibit reads: Across all states, subjects, and grade levels, CSRD schools made gains relative to non-CSRD 
schools in 55 percent of the states, on average, and relative declines in 43 percent of the states. 
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This report provides a preliminary analysis covering a time period during which CSRD 
schools had had limited time to implement their programs. 

Overall, these data show little evidence of a positive impact of the CSRD program, at this 
early stage, on student achievement on state assessments. In states where CSRD schools made 
significant gains, so too did non-CSRD schools. There are some positive indications in the data, 
including the findings that states are more likely to show gains than declines in the within-state 
rankings of their CSRD schools, although these changes were not statistically significant. The 
gains were more prevalent in states that have assessment trend data available for a longer period 
of time (three years instead of the usual two years), raising hope for future analyses when 
additional years of data are available. 

Once again, it is important to keep in mind that this is a very preliminary analysis, 
because the time frame covered in this report is too short to expect large effects of the CSRD 
program. At the time recorded by the test data points, most schools had been receiving CSRD 
funding for less than one or two full school years. 

Further analysis is needed as additional years of state assessment data become available 
in order to understand the impact of the CSRD program on student achievement. Additional 
time would allow for deeper implementation of reforms. Additional time would also provide 
increased data points, allowing for a more robust analysis. A later report on student achievement 
in the first cohort of CSRD grantees would offer a better understanding of the progress of 
schools participating in this program. 
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Appendix A 

Methodology for Achievement Analyses 



This appendix outlines the methodology the RAND Corporation used to conduct the achievement 
analysis. In order to understand the evaluation questions outlined above, separate state-by-state analyses 
of CSRD and non-CSRD schools were performed. Although the detailed analyses are not reported here, 
the state-specific analyses encompassed the following tasks: 

• First, a profile of each state’s accountability system was developed in order to document the data 
contained in the National Longitudinal School-Level State Assessment Score Database as of 
January 2002 and understand the testing regime over the time period covered by the study 

(SY 1998-99 through SY 2000-01). 1 

• Second, a comparison group of non-CSRD schools that were similar to the CSRD schools in 
terms of poverty status was defined. Because the CSRD program emphasized selecting schools 
identified in need of improvement, these schools are likely to be both higher poverty and lower 
achieving than schools in the rest of the state. Therefore, to understand the gains made by these 
schools, it is important to compare them with the performance and the gains made by schools that 
match them in terms of poverty, where possible. Because analysts may disagree with our choice 
of comparison schools, the analyses were bound by comparing the CSRD schools to a set of 
comparison schools as well as the to entire set of non-CSRD schools in the state. 

• Third, the average level of performance in CSRD schools was examined to see whether and how 
it differed from that of non-CSRD comparison schools and all non-CSRD schools. Changes in the 
average level of performance among CSRD schools from the baseline year to one or two years 
after receipt of CSRD funds were compared to see whether these schools improved over time. 
Changes in the average level of performance of comparison schools and all non-CSRD schools in 
the state also were calculated. 

• Fourth, CSRD schools were ranked in two distributions: one of comparison non-CSRD schools 
and the other of all non-CSRD schools in the state, and gains made by CSRD schools relative to 
these two distributions were examined. 

• Fifth, the gains in all schools and all states in the average level of performance in CSRD schools 
and in the rankings of CSRD schools relative to comparison non-CSRD schools and all non- 
CSRD schools were summarized. 

These tasks are further detailed below. 



Profile Each State’s Testing Regime 

Analysts at RAND developed a profile to examine the test used for accountability purposes, the grades 
and subjects tested and the metric in which they reported scores. The profile included detailed notes 
regarding changes in tests or test metrics over time, the nature and extent of accommodations, and a li nk 
to the state’s Web site. This allowed us to examine whether the reported scores were comparable over 
time. 



1 This information was not present in the database. 
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Two examples illustrate why this task was an important and necessary part of the analysis. Mississippi 
adopted a new criterion-referenced assessment system, the Mississippi Curriculum Tests, in 2000-01. 
Thus change scores could not be computed for Mississippi schools (Exhibit A-l). 



Exhibit A- 1. State Profile of Testing Regime: Mississippi 



Test on AIR Database 

TerraNova, norm-referenced exam 
Writing Assessment 

Iowa Test of Basic Skills (ITBS), norm-referenced exam 

Year Tests First Administered 

1999-2000 - TerraNova 
1994-95 -ITBS 

Time of Test Administration 

Spring - TerraNova 
Fall -ITBS 



Test Scores on AIR Database 



Grades 


Test 


Subjects 


Years 


Reported Test Metric 


3rd - 8th 


TerraNova 


Reading; language arts; 
mathematics; total score 


1999-2000 


Percentile 


4th, 7th 


Writing 

Assessment 


Writing 


1999-2000 


Percent at levels 


4th-9th 


ITBS 


Reading; language arts; 
mathematics 


1994-95; 1995-96; 
1996-99; 1997-98; 
1998-99 


Mean NCE 



Data Notes 

• Between the fall of 1994 and 1998, Mississippi administered the ITBS in grades four through eight and the 
Tests of Achievement and Proficiency in grade nine. Subject area tests and the functional Literacy Examination 
were used to determine high school level proficiency. 

• In the summer of 1999, Mississippi began the design and approval process for the new assessment system that 
includes the Terra Nova and criterion-referenced assessments. The criterion-referenced assessments 
(Mississippi Curriculum Tests) were first administered in 2000-01. Mississippi warns against calculating 
change scores across the two tests. 

• For students with disabilities, the IEP committee determines whether the student will participate in the 
assessment. Scores will be excluded from school-level reporting for students who receive accommodations that 
exceed those allowed for a specific test, students who are provided instruction utilizing an alternate or parallel 
curriculum in any of the assessed areas, or students whose IEPs indicate that due to educational delay they are 
not expected to meet the basic curriculum objectives for that year. 

• Limited English proficient (LEP) students are expected to participate in the assessments, but may be exempted 
for no more than two years based on a language assessment designed for LEP students. 

• The scores of LEP students who participate in the assessment are included in the overall school score unless the 
student met the requirements to be exempted but elected to participate in the assessment or if the appropriate 
accommodations for the assessment were insufficient or inappropriate to meet a student’s needs. 

State Accountability Web Site: www.mde.kl2.ms.us/acad/osa/testdata.html. 
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In Utah, although the test had not changed, the Salt Lake City School District changed their practices for 
exempting limited English proficient students in 2000-01 (Exhibit A-2). Thus, the 2000-01 results could 
not be compared with those of earlier years. For this reason, we excluded schools in this district from our 
analyses of Utah schools. 



Exhibit A-2. State Profile of Testing Regime: Utah 



Test on AIR Database 

Stanford-9, norm-referenced exam 

Year Test First Administered 

1997-98 

Time of Test Administration 
Fall 



Test Scores on AIR Database 



Grades 


Subjects 


School Years 


Reported Test Metric 


3rd 


Reading; mathematics; language; 
science; social studies 


2000-01 


Percentile 


5th, 8th, 11th 


Reading; mathematics; language; 
science; social studies; total battery 


1997- 98; 

1998- 99; 

1999- 2000; 

2000- 01 


Percentile 



Data Notes 

• Utah developed a basic skills test to be administered as an exit exam to lOth-grade students starting in 2003. 

In addition, districts administer criterion-referenced subject tests to all grades as part of the Core Assessment 
Program, but these scores are not included in the AIR database. 

• Students with disabilities may be exempted from testing if their IEP team determines that the student cannot 
demonstrate his or her knowledge without an accommodation. 

• Limited English proficient (LEP) students may be exempted from testing only if they have received less than 
three years of instruction primarily in English and cannot demonstrate their knowledge in English. 

• In 2000-01, the Salt Lake City School District changed their practices for exempting LEP students in a way that 
was inconsistent with the state requirements. In many cases, students were exempted who had not reached the 
“fluent/competent” level on the IPT Language Proficiency Tests, regardless of how long the student had 
received in English instruction. This resulted in the exemption, in some schools, of up to 75 percent of the 
students. The exemption of additional students from Salt Lake School District alters the data trend that has 
been established and makes the 2000-01 results noncomparable. 

State Accountability Web Site: www.usoe.kl2.ut.us/eval/. 
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Analyzing the Level of Performance in CSRD Schools 

As mentioned earlier, states used different metrics for reporting test scores, some of which lend 
themselves to aggregation across sets of schools, while others do not. In addition, some states had no 
school-level test score information. Exhibit A-3 shows the states that fell into each category. 

Exhibit A-3. Metrics Used by States in Reporting School-Level Test Scores 



Test Score Metric 


States 


Scale score or normal curve 
equivalent (NCE) score 


CA, DE, FL, IL, KY, MA, ME, MN, MS, NC, NH, NJ, NY, PA, SC 


Percent passing level 


AK, CO, CT, DC, HI, IN, KS, LA, MD, MI, MO, OH, OK, OR, RI, TX, VA, VT, 
WA, WI, WY 


Percentile 


AL, AR, AZ, GA, ID, MS (post-baseline year), NV, SD, TN, UT, WV 


No uniform, school-level test 
scores 


IA, MT, ND, NE, NM 



If the test metric allowed aggregation across sets of schools (scale scores/NCEs, percent passing) 

When the data allowed us to aggregate across schools, data were summarized in a table that provides a 
description of the average performance levels in the original metric (for example, percent passing) to 
allow the reader to examine the average performance levels for CSRD schools, a comparison group of 
non-CSRD schools, and all non-CSRD schools in the entire state. The table includes sample sizes and 
provides an indication of how reliable the estimate was. 

Simple two-sample t-tests were conducted to determine whether the average level of performance of 
CSRD schools differed from that of comparison schools, and that of all non-CSRD schools in the state for 
each school year, using a p-value of 0.05 to determine statistical significance. The test for levels is useful 
to determine if there are significant differences between the two groups at a point in time. 

In the tables accompanying each state summary, only results for data aggregated across grades for each 
school level are shown. Thus, although the detailed analyses examined means and changes in means for 
all grades for which information was available, the summary tables show only the results for elementary, 
middle, and high schools. When data were available, for elementary schools, third- through fifth-grade 
scores were averaged; for middle schools, seventh- and eighth-grade scores were averaged; and for high 
schools, ninth- through 1 l th -grade scores were averaged. In some states only a particular grade was tested, 
in which case that score was reported for the school as a whole. Each table shows the grades on which the 
reported score was based. 

Exhibit A-4, which shows data for California schools, reported data as scale scores for four years, 

SY 1997-98 to SY 2000-01. There were 47-49 CSRD elementary schools (depending on the year), 13 
middle schools, and 1 0 high schools. As mentioned above, third- through fifth-grade scores were 
averaged to get a score for each elementary school, seventh- and eighth-grade scores to get a score for 
each middle school; and ninth- through 1 lth-grade scores to get a score for each high school. School 
scores were then averaged to get a mean for CSRD schools, a mean for the comparison group, and a mean 
for all non-CSRD schools in the state (as well as all schools). 

Looking at the baseline year, SY 1998-99, the mean for CSRD elementary schools in reading was 603 
compared with 613 for the comparison group and 629 for all non-CSRD schools. By 2000-01, the means 
for each group had increased to 609 (CSRD schools), 619 (comparison group), and 635 (all non-CSRD 
schools). Data for middle and high schools can be interpreted in a similar fashion. 
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Exhibit A-4. School-Level Achievement in California Elementary Schools: 
SY 1997-98 through SY 2000-01, Average Scale Scores 







CSRD 

Schools 






High-Poverty 
Non-CSRD Schools 






All 

Non-CSRD Schools 






All 

Schools 






1997- 

1998 


1998- 

1999 


1999- 

2000 


2000- 

2001 


1997- 

1998 


1998- 

1999 


1999- 

2000 


2000- 

2001 


1997- 

1998 


1998- 

1999 


1999- 

2000 


2000- 

2001 


1997- 

1998 


1998- 

1999 


1999- 

2000 


2000- 

2001 


















Average Scale Score 














Elementary Schools 
(3rd, 4th, & 5th 
Grades) 


































Reading 


598 


603 


607 


609 


609* 


613* 


616* 


619* 


626* 


629* 


632* 


635* 


626 


629 


632 


634 


(Number of schools) 


(n=47) 


(n=48) 


(n=47) 


(n=49) 


(n=2405) (n=2457) 


(n=2438) 


(n=2477) 


(n=4522) 


(n=4619) 


(n=4578) 


(n=4734) 


(n=4569) 


(n=4667) 


(n=4625) 


(n=4783) 


Mathematics 


593 


601 


608 


612 


603* 


608* 


616* 


620* 


616* 


622* 


629* 


633* 


616 


622 


629 


633 


(Number of schools) 


(n=47) 


(n=48) 


(n=47) 


(n=49) 


(n=241 1) (n=2460) 


(n=2439) 


(n=2479) 


(n=4535) 


(n=4622) 


(n=4578) 


(n=4741) 


(n=4582) 


(n=4670) 


(n=4625) 


(n=4790) 


Middle Schools 
(7th & 8th Grades) 


































Reading 


657 


662 


664 


665 


666* 


667 


669 


670 


679* 


681* 


682* 


683* 


679 


681 


682 


683 


(Number of schools) 


(n=13) 


(n=13) 


(n=13) 


(n=13) 


(n=730) 


(n=753) 


(n=745) 


(n=761) 


(n=1665) 


(n=1703) 


(n=1663) 


(n=1777) 


(n=1678) 


(n=1716) 


(n=1676) 


(n=1790) 


Mathematics 


655 


661 


663 


662 


661 


664 


666 


668 


673* 


675* 


678* 


680* 


673 


675 


678 


680 


(Number of schools) 


(n=13) 


(n=13) 


(n=13) 


(n=13) 


(n=732) 


(n=753) 


(n=745) 


(n=761) 


(n=1665) 


(n=1704) 


(n=1659) 


(n=1780) 


(n=1678) 


(n=1717) 


(n=1672) 


(n=1793) 


High Schools 
(9th, 10th, & 11th 
Grades) 


































Reading 


671 


673 


675 


673 


675* 


675 


677 


676 


688* 


688* 


689* 


688* 


688 


688 


688 


688 


(Number of schools) 


(n=10) 


(n=10) 


(n=10) 


(n=10) 


(n=236) 


(n=241) 


(n=242) 


(n=242) 


(n=l 120) 


(n=1137) 


(n=l 1 10) 


(n=1186) 


(n=l 130) 


(n=l 147) 


(n=l 120) 


(n=l 196) 


Mathematics 


677 


680 


685 


684 


680 


682 


683 


683 


690* 


691* 


693* 


693* 


690 


691 


693 


693 


(Number of schools) 


(n=10) 


(n=10) 


(n=10) 


(n=10) 


(n=238) 


(n=241) 


(n=243) 


(n=244) 


(n=l 118) 


(n=l 143) 


(n=l 1 12) 


(n=1188) 


(n=l 128) 


(n=l 153) 


(n=l 122) 


(n=l 198) 



Note: * indicates a statistically significant difference between the mean score for the particular group (i.e., high-poverty non-CSRD schools or all non-CSRD 
schools) and mean score for the CSRD schools (p-value<0.05). 
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To determine how the average performance of CSRD schools compared with that of similar non-CSRD 
comparison schools and all other non-CSRD schools in the state, simple two sample t-tests were 
conducted, using a p-value of .05 to judge statistical significance. Statistically significant results are 
indicated by an asterisk. In California: 

• At the elementary level (grades 3-5 combined), CSRD schools scored significantly below 
comparison schools (high-poverty non-CSRD schools in the state) in both subjects across the four 
years (SY 1997-98 through SY 2000-01). 

• At the middle and high school levels, CSRD schools scored significantly below comparison 
schools in SY 1997-98, but there was no difference in the average performance of CSRD and 
comparison schools from SY 1998-99 to SY 2000-01. 

• At the elementary, middle, and high school levels, CSRD schools scored significantly lower than 
all non-CSRD schools in both reading and mathematics and across all four years (SY 1997-98 
through SY 2000-01). 

If the test metric did not allow aggregation across sets of schools (percentile ranks) 

The most common method of ensuring comparability of scores across different tests or aggregating across 
schools is to calculate a z-score for each school that tells us how far a raw score is from the mean of the 
distribution of the state schools, the distance being expressed in standard deviation units. However, the z- 
score assumes that the underlying distribution is normal or nearly so. In some states, given the metric 
used and the test, this assumption is not likely to be met. In some states, the small sample size of the 
CSRD schools may make the mean subject to considerable measurement error. Thus, a method was 
employed that has the advantages of being distribution- free. 

First, all schools within a given distribution were ranked. All schools within the overall state distribution 
were ranked to examine the level of performance in CSRD and non-CSRD schools. Essentially, the 
procedure ranked values (i.e., percentile rank) from lowest to highest, assigning the ra nk 1 to the school 
with the lowest score, 2 to the next higher, and so on up to ra nk n, the number of schools in the 
distribution. Tied values were given averaged ranks. 

Next, the rank r, for each school i, was divided by (n+1), to get values in the range 0 to 1; these values 
estimate the cumulative distribution function. An inverse Gaussian cumulative distribution function was 
then applied to these fractional ra nks to obtain standardized scores. In particular, a value was estimated 
for each school — call it a standardized unit, m, to distinguish it from a z-score — as follows: 

Ui= O 1 (r/(n+l)) 

where ® 1 is the inverse cumulative normal (e.g., probit) function, r, is the ra nk of the i th school, and n is 
the total number of schools in the distribution. It has the property that it gives somewhat greater weight 
to increases in the tails of the distribution than to increases in the middle, which is important when 
examining CSRD schools because so many of them are low-performing. 

The values of the «,s range from approximately -3 to +3 and are analogous to z-scores in terms of 
showing how far each school is from the center of the distribution. However, they avoid the assumption 
that the underlying distribution of scores is normal. If the underlying distribution is normal, they will 
essentially be equal to z-scores. The «,s can be averaged across schools, even when the original metric 
(such as percentile rank) does not lend itself to averaging across a set of schools. 
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The M/S were averaged, yielding a mean score for CSRD schools, comparison non-CSRD schools, and all 
non-CSRD schools in the state. Simple two-sample t-tests were conducted to determine whether the 
average performance in these schools differed from each other. 

These data are summarized in tables that are part of the state-specific summaries. The tables, as 
mentioned earlier, show data at the school level, aggregated across grades. 

Exhibit A-5, which shows a state that reports test scores in percentile ranks, presents standardized scores 
for Alabama for SY 1998-99 and SY 1999-2000 (the only two years for which data were available on the 
AIR database). Thus, instead of reporting an average for the CSRD schools in terms of percentile ranks, it 
reports an average in terms of standardized units. CSRD schools are nested within the distribution of all 
schools in the state. The mean score for all schools is by definition equal to zero. The mean score for all 
non-CSRD schools is very close to zero because they represent a large proportion of all the schools in the 
state; thus the mean on a standardized scale for these schools must be close to the overall mean of the full 
distribution. The mean score for CSRD schools represents their standing relative to all schools in the 
state; similarly the mean score for the comparison schools represents their standing relative to all schools 
in the state. Exhibit A-5 shows that: 

• Elementary CSRD schools scored significantly lower than both the highest-poverty non-CSRD 
schools and all non-CSRD schools in both subjects and across both years. For example, in 1998- 
99, CSRD schools scored 1.4 standardized units below all other schools in the state in reading, 
significantly below the average for the comparison group (1 standardized unit below all other 
schools in the state), and significantly below the 0.0 mean for all non-CSRD schools in the state. 
While the highest-poverty schools also rank lower in the overall distribution, they tend to ra nk 
higher than CSRD schools. 

• At the middle school level, both the comparison and CSRD schools ranked about 1.0 standardized 
units below the overall mean for the state in both subjects. CSRD schools scored significantly 
below non-CSRD schools in both subjects and in both years but are similar to the comparison 
schools. 

• At the high school level, the CSRD schools are between 0.9 - 1.4 standardized units below the 
overall mean for the state, while comparison schools are between 1.1 - 1.6 standardized units 
below. CSRD schools scored significantly below non-CSRD schools in both subjects and in both 
years but are similar to the comparison schools. 
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Exhibit A-5. School-Level Achievement in Alabama Schools: 
SY 1998-99 through SY 1999-2000, Standardized Scores 





CSRD 

Schools 


Highest-Poverty 
Non-CSRD Schools 


All 

Non-CSRD Schools 


All 

Schools 




1998- 

1999 


1999- 

2000 


1998- 

1999 


1999- 

2000 


1998- 

1999 


1999- 

2000 


1998- 

1999 


1999- 

2000 




Standardized Scores 


Elementary Schools 
(3rd, 4th, & 5th Grades) 

(Number of schools) 
Reading 

(Number of schools) 
Mathematics 


(n=24) 

-1.4 

(n=25) 

-1.3 


(n=24) 

-1.4 

(n=24) 

-1.2 


(n=136) 

-1.0* 

(n=137) 

-0.7* 


(n=138) 

-1.0* 

(n=138) 

-0.8* 


(n=646) 

0.0* 

(n=648) 

0.0* 


(n=644) 

0.0* 

(n=644) 

0.0* 


(n=670) 

0.0 

(n=673) 

0.0 


(n=668) 

0.0 

(n=668) 

0.0 


Middle Schools 
(7th & 8th Grades) 

(Number of schools) 
Reading 

(Number of schools) 
Mathematics 


(n=14) 

-1.1 

(n=14) 

-0.9 


(n=15) 

-1.2 

(n=15) 

-1.0 


(n=57) 

-1.2 

(n=58) 

-1.0 


(n=62) 

-1.2 

(n=62) 

-1.0 


(n=463) 

0.0* 

(n=464) 

0.0* 


(n=463) 

0.0* 

(n=463) 

0.0* 


(n=477) 

0.0 

(n=478) 

0.0 


(n=478) 

0.0 

(n=478) 

0.0 


High Schools 
(9th, 10th, & 11th 
Grades) 

(Number of schools) 
Reading 

(Number of schools) 
Mathematics 


(n=17) 

-1.4 

(n=17) 

-1.1 


(n=17) 

-1.2 

(n=17) 

-0.9 


(n=23) 

-1.6 

(n=24) 

-1.1 


(n=27) 

-1.5 

(n=27) 

-1.2 


(n=343) 

0.1* 

(n=349) 

0.1* 


(n=354) 

0.0* 

(n=354) 

0.0* 


(n=360) 

0.0 

(n=366) 

0.0 


(n=371) 

0.0 

(n=371) 

0.0 



Note: * indicates a statistically significant difference between the mean score for the particular group (i.e., highest- 
poverty non-CSRD schools or all non-CSRD schools) and mean score for the CSRD schools (p-value<0.05). 



Analyzing the Change in Level of Performance over Time in CSRD Schools 
Relative to Comparison Group and All Non-CSRD Schools 

Change in performance over time was analyzed in three ways: 

A. For states in which data were reported in original units that can be aggregated (average scale 
score, mean NCEs, or percent passing) 2 

For each CSRD school, comparison school, and non-CSRD school in the state, a simple change score was 
computed from baseline to one to two years after receipt of CSRD funds. The key questions were whether 
the CSRD schools made any gains (or losses) over time, whether these changes were statistically 
significant, and whether comparison schools and all non-CSRD schools experienced similar changes in 
test scores. The small sample sizes for CSRD schools in most states lead to two problems: the average 
scores are themselves subject to measurement error, and it is harder to find statistically significant gains 
for CSRD schools. 



2 ln states that reported percentile ranks, gain scores were not computed using the ranks. This analysis was focused 
on the gain scores of schools. For example, did the average scale score of CSRD schools increase over time? In the 
next section, the transformed metric was used to examine the question of whether the CSRD schools improved their 
relative standing compared with similar schools. 
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There are three points that should be noted about the gain scores. First, for CSRD schools, the gain score 
represents different “post-CSRD” periods depending on the month the schools received their funds. 
Second, in some states, gains are measured over a one-year period, from baseline to one year after receipt 
of CSRD funds (SY 1998-99 to SY 1999-2000) and in others, over a two-year period (SY 1998-99 to 
SY 2000-01). Third, even if differences in gains were evident across the maximum of two years, it 
would be difficult to attribute those gains to the CSRD program. Ideally, one would want to control for 
the length of time for which these schools received CSRD funding and also for the length of time actually 
implementing the components of reform called for by CSRD. For example, some schools received CSRD 
awards as late as November 1999; for these schools, the spring 2000 test score does not even represent a 
full year after receipt of funding. Thus these schools have had very little time to implement reform, much 
less see the effect of those reforms. All that can be measured in this analysis is whether CSRD schools 
made gains over this particular one- or two-year period and whether comparison schools and all non- 
CSRD schools made similar gains. 

Returning to the California example, shown in Exhibit A-4, the significance of the change in scale scores 
was tested separately for the CSRD, non-CSRD comparison schools, and all non-CSRD schools that had 
scores in SY 1998-99 and SY 2000-01. 

• Over this two-year period, the CSRD elementary schools made statistically significant increases 
of six points in reading, and these gains were similar to those made by the comparison group and 
all non-CSRD elementary schools. Similarly, in mathematics, California elementary schools 
made statistically significant gains of 1 1 points, which were similar to the 11-12 point gains made 
by comparison and all non-CSRD schools in the state. 

• The CSRD middle schools made gains of three points in reading and one point in mathematics, 
and these were equal to or smaller than those made by non-CSRD comparison schools (three 
points in reading and four points in mathematics). All non-CSRD middle schools made gains of 
two points and five points in reading and mathematics respectively. None of these changes was 
statistically significant. 

• The CSRD high school scores remained the same in reading, as did the scores of comparison and 
all non-CSRD high schools. CSRD high school scores increased by four points in mathematics; 
this was larger than the one- to two-point increase experienced by comparison and all non-CSRD 
schools. None of the changes was statistically significant. 

B. In order to test whether CSRD schools had improved relative to comparison schools, all states 
for which change scores could be calculated: 

CSRD schools were ranked in the distribution of non-CSRD comparison schools, using the score for the 
individual school (either in the original metric or the transformed metric) and calculated u , -s as explained 
above. Because of the variability in the data across grades, the focus was only on the aggregated test score 
for the school (i.e., averaged scores across grades). Gain scores in the «, metric, 6„ were then calculated 
for each school in the two groups (CSRD schools and comparison non-CSRD schools). The change was 
calculated from the baseline year, SY 1998-99, to either SY 1999-00 or SY 2000-01, depending on 
availability of data. A one-tailed t-test was used to examine the hypothesis that the relative gains of the 
CSRD schools (relative to the comparison group) were significantly greater than zero. In other words, did 
the CSRD schools move up in the distribution of comparison and CSRD schools. 
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The null and alternative hypotheses follow the following notational form: 



H 0 : S,<0 
H a : 8, > 0 

The null hypothesis is one of no difference (8) between the standardized mean scores against a one-sided 
alternative that CSRD schools gained relative to the comparison group. 3 Significant differences (p- 
values<.05) are noted with an asterisk in the tables. 

C. In order to test whether CSRD schools had improved relative to all non-CSRD schools, all states 
for which change scores could be calculated: 

Because the choice of a comparison group potentially could bias results in favor of finding an effect, our 
estimates were bounded by ranking CSRD schools in the distribution of all non-CSRD schools and 
determining whether CSRD schools had improved relative to these schools. The procedure was exactly 
the same as the one outlined in the above section. Once the changes were calculated, a similar one-tailed 
test was conducted to see whether the gains of the CSRD schools (relative to all non-CSRD schools) were 
significantly greater than zero. Obviously, the same caveats listed above apply to this analysis. 

Example 

The examples of California and Alabama illustrate the analyses described in sections B and C above. 
Exhibits A-6 and A-7 show the change scores for California and Alabama CSRD schools relative to the 
comparison schools (high-poverty non-CSRD schools in California and highest-poverty non-CSRD 
schools in Alabama) and all non-CSRD schools. Note that the change score for California schools 
represents change over a two-year time period (SY 1998-99 to SY 2000-01) while the change score for 
Alabama represents a one-year change (SY 1998-99 to SY 1999-2000). 



California (Exhibit A-6): 

Relative to comparison schools (high-poverty non-CSRD schools): 

• The average elementary CSRD school score declined by 0.09 standardized units in reading and 
by 0.04 standardized units in mathematics. 

• At the middle school level, the average CSRD school score declined 0.05 standardized units in 
reading and 0.17 standardized units in mathematics. 

• At the high school level, the average CSRD high school score remained essentially unchanged in 
reading but increased by 0. 19 units in mathematics. 

• However, none of these changes was statistically significant from zero; i.e., CSRD schools did 
not make significant gains relative to comparison schools in California. In statistical terms, this 
result fails to reject the null hypothesis that the average change in CSRD schools relative to the 
comparison non-CSRD schools over this time period was essentially zero. 



3 When there was only one CSRD school at a grade level in a state, this test could not be run. A single value could 
not be tested for statistical significance against zero. 



A-10 




Exhibit A-6. School-Level Achievement Changes in CSRD Schools 
Relative to Comparison Non-CSRD Schools and Relative to All Non-CSRD Schools in California, 

SY 1998-99 to SY 2000-01 





Standardized Scores 




Relative to Comparison 
Non-CSRD Schools 


Relative to All Non- 
CSRD Schools 


Elementary Schools 
(Grades 3-5 Combined) 






(Number of schools) 


(n=48) 


(n=48) 


Reading 


-0.09 


-0.06 


Mathematics 


-0.04 


-0.02 


Middle Schools 
(Grades 7-8 Combined) 






(Number of schools) 


(n=13) 


(n=13) 


Reading 


-0.05 


-0.03 


Mathematics 


-0.17 


-0.12 


High Schools 
(Grades 9-11 Combined) 






(Number of schools) 


(n=10) 


(n=10) 


Reading 


-0.03 


0.01 


Mathematics 


0.19 


0.11 



Exhibit reads: On average, relative to high-poverty non-CSRD schools, CSRD elementary 
schools’ reading scores decreased by 0.09 standardized units from SY 1998-99 to SY 2000-01. 

Relative to all non-CSRD schools: 

• The average elementary CSRD school score declined by 0.06 standardized units in reading and 
remained essentially unchanged in mathematics. 

• At the middle school level, the average CSRD school score declined 0.03 standardized units in 
reading and 0.12 standardized units in mathematics. 

• At the high school level, the average CSRD high school score remained essentially unchanged in 
reading but increased by 0. 1 1 units in mathematics. 

Again, none of these changes was statistically significant from zero; i.e., CSRD schools did not make 

significant gains relative to all non-CSRD schools in California. 

Alabama (Exhibit A-7) 

Relative to comparison schools (highest-poverty non-CSRD schools): 

• At the elementary level, the average score in CSRD schools remained essentially the same in 
reading and increased by 0. 17 standardized units in mathematics over this one-year time period. 

• CSRD middle schools did somewhat worse compared with comparison schools, with a decline of 
0.34 standardized units in reading and 0.21 standardized units in mathematics. 
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• CSRD high schools remained at the same level relative to comparison schools over this time 
period. 

However, none of these changes were statistically significant; i.e., CSRD schools did not make significant 
gains relative to comparison schools in Alabama. That is, they fail to reject the null hypothesis that the 
average change in CSRD schools relative to the comparison schools over this time period was zero. 

Relative to all non-CSRD schools: 



• At the elementary level, the average score in CSRD elementary schools remained essentially the 
same in reading and increased by 0.17 standardized units in mathematics over this one-year time 
period. 

• CSRD middle schools did somewhat worse compared with all non-CSRD schools, with a decline 
of 0. 16 standardized units in reading and 0. 1 8 standardized units in mathematics. 

• CSRD high school scores increased by 0.08 standardized units in reading and 0.12 standardized 
units in mathematics. 

• However, none of these changes was statistically significant from zero; i.e., CSRD schools did 
not make significant gains relative to all non-CSRD schools in Alabama. 

Exhibit A-7. School-Level Achievement Changes in CSRD Schools 
Relative to Comparison Non-CSRD Schools and Relative to All Non-CSRD Schools in Alabama, 

SY 1998-99 to SY 1999-2000 





Standardized Scores 




Relative to Comparison 
Non-CSRD Schools 


Relative to All 
Non-CSRD Schools 


Elementary Schools 
(3rd, 4th, & 5th Grades) 






(Number of schools) 


(n=22) 


(n=22) 


Reading 


-0.03 


-0.03 


Mathematics 


0.17 


0.17 


Middle Schools 
(7th & 8th Grades) 






(Number of schools) 


(n=14) 


(n=14) 


Reading 


-0.34 


-0.16 


Mathematics 


-0.21 


-0.18 


High Schools 
(l()th Grade) 






(Number of schools) 


(n=17) 


(n=17) 


Reading 


-0.03 


0.08 


Mathematics 


0.00 


0.12 



Exhibit reads: On average, relative to the highest-poverty non-CSRD schools, elementary CSRD 
schools’ reading scores decreased by 0.03 standardized units from SY 1998-99 to SY 1999-2000. 
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